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determine why differences exist for older and Hispanic students. 
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ERIC 



College of the Canyons 



Disproportionate Impact Study 
Writing Sample Test 

This research is designed to provide preliminary data in monitoring for 
disproportionate impact of placement into basic skills courses for various demographic 
groups. 

The study is designed to answer the following research question: Does placement 
into different levels of basic skills courses differ significantly for students in particular age, 
gender, ethnic or disability groups based on an assessment instrument, method or 
procedure? 



Method 



Design 

Tested Population.' SXudiQnXs w'ho took the APS Writing Test and the Writing Sample 
Test for Fall 1995 and Spring 1996 enrollment were included. Only those students who 
scored on or one point above or below the APS Writing Test cut scores were included in 
the tested population. These are the only students for whom the Writing Sample Tests are 
read and scored. They are also the only group for which the results of the Writing 
Sample Test are used in the placement recommendation process. A total of 617 students 
were included in the tested population. 

Eligible Population. Using the "recommended placement" made by the English faculty 
after reading a student's Writing Sample Test, students were divided into four groups, 
those recommended for English 01 1, English 035, English 090 and English 101. See the 
Essay Evaluation Guide on page 4 which provides descriptions of the level of competence 
expected for each course placement recommendation. 

Special Population Groups. We obtained demographic information from the CAPP score 
sheets for each student. All of the 61 7 students in the tested population indicated their 
age and gender; 607 indicated their race/ethnicity and disability. 
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Resuhs 



Analysis of the Data 

The analysis was conducted by comparing the percent of students in the special 
groups for the tested population to the percent of students in special groups who were 
recommended to enroll into each of the four English courses based on the Writing Sample 
Test recommended placement. 

We examined the Writing Sample Test placement recommendations for age, 
gender, race/ethnicity and disability, applying the Equal Employment Opportunity 
Commission 80% guideline related to hiring practices. The EEOC 80% rule is used to 
determine if an investigation of the procedures should be undertaken. The 80% rule is 
calculated by forming a ratio between the percent of students in each group who would be 
placed according to the cut score. For example, in Table 1, where females are used as the 
majority group, the percent of male students eligible/tested (.0393) is divided by the 
percentage of female students eligible/tested (.0356). The result is 1 10.4%, well above the 
80% rule cut-ofF. If fewer thnn 80% of the minority group students would be placed, it 
indicates that the placement scheme should be reviewed to determine if discriminatory 
practices or a biased test are being employed. 



Findings 

Gender 



Disability 



Age 



Race/Ethnicity 



No significant disproportionate placement found. See 
Tables 1 through 4. 

Cell sizes for tested students fell below 20, making them too 
small to make a determination. See Tables 5 through 8. 

Significant disproportionate placement found for older 
students (age 25 and above) for English 090. The 
proportion of older students to be placed falls slightly below 
the 80% cut-off (76.1%). See Tables 9 through 12. 

Significant disproportionate placement found for Hispanic 
students for English 101. See Tables 13 through 16. 



Discussion 

While disproportionate impact was found in only two of the 36 possible course 
levels and special population groups, the college is concerned about the findings. 
Practices, which are in accord with the state matriculation regulations, are in place to 
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ensure that the placement recommendation from this one placement test isn't the only 
measure used to recommend course placement in English. Multiple measures are used by 
college counselors, including, but not limited to, the score a student receives on the APS 
Writing Test, grade received in last high school English course, recency of last high school 
English course, and overall high school GPA. Students are reminded that placement 
advice is advisory, not mandatory. Finally, students are encouraged to seek additional 
advice from counselors or faculty members if they desire it. Through these practices, the 
college has reduced to a minimum any possible harm from the possible disproportionate 
impact of the Writing Sample Test itself 



Future Research 

Further research is needed to determine why differences exist between certain 
groups. Aje the differences related to the nature of the instrument or are they due to 
socioeconomic or other factors? 

A study of differential prediction may be helpful in determining whether the 
Writing Sample Test predicts differentially for certain groups of students. 



DISPWTUT.WTS 

Office of Institutional Development 
October 1996 
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ESSAY EVALUATION GUIDE 



This information is provided to help Counselors & Program Advisors interpret the essay score. 
Course Recommendation; English 101 Score; 7 

A level 7 English 101 essay wUl be fluent and clearly organized. The thesis is clearly stated 
and well supported. The essay shows a sophisticated command of language and sentence 
variety and is free of errors in sentence structure, grammar, and mechanics. It displays 
originality and depth of thought. 



Course Recommendation; English 101 Score; 6 

A level 6 English 101 essay will have clarity of expression and organization. The thesis is 

of suffiden^ 'r are developed, and the essay is 

ot suflicient length to demonstrate control of the essay process. The essay shows a good 

comman o anguage and sentence variety and is free of all but a few errors in sentence 
structure, grammar, and mechanics. 

Course Recommendation; English 090 Score- 5 

A level 5 English 090 essay will be less fluent and organized than a level 6. There is an 

Therek paragraphing may be unevenly developed . 

J I? awkwardness in syntan. The essay may 

have some problems m diction, grammar, mechanics, and sentence structure. 

Course Recommendation; English 090 Score- 4 

A level 4 English 090 essay will be less organized and fluent than a level 5 The thesis if 
n~r' of the essay. Some organization is apparent! but ’ 

levelTe-sav^p'^d^^ underdeveloped or repetitive. There is less variety in sentences than the 
^ict on ^°^^,^''’kwardness in syntax. The essay usually has some problems in 

iadabilU\ ’ sentence structure, but these do not interfere with 



Course Recommendation; English 035 Score; 3 

A level 3 English 035 essay will fit one or more of these categories; 

It may have poor organization; the thesis, if present, may not be developed- ‘ 
paragraphs may be minimal or rambling; it may have confused syntax, fragments, or run-on 
hPfrin f ' f 7^^ diction, grammar, and mechanics, which 

teS to be srorterThSiTS^^s"^^^ “ development, but wiU 



Course Recommendation; English 035 Score- 2 

noI"e'iD„t"en"the?ie "dll fit one or more of the foUownng categories; organization may 
paraiantoro; or too vague for development; it may show no grasp of 

framSn^s nr' 1>P rppdom; it may have confused syntax or excessive 

fragments or run-on sentences; it may have numerous, serious errors in diction grammar or 
mechanic, which make the essa y difficult to understand; and /or it wiU ramble associatively. 

Course Recommendation; English Oil Score- 1 

onlyrfew!ha«fc‘’s!V categories: it may contain 

fTroln chaotic sentences; the sentence structure and/or the syntax will be garbles- its 

SdreTs^tii^pJo^pr’ unreadable; and/or it ^nll not 
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Table 1. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Gender into English Oil 

80% Rule Index: I'emales used as majority group 



Gender 


Tested 

Population* 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Female 


337 54.6% 


12 


52.2% 


-2.4% 


.0356 


Male 


280 45 4% 


ii 


47,8% 


+2.4% 


0.393 


TOTAL 


617 . 100.0% 


23 


100.0% 







80% Rule Index: 1 10.4% 



' ■ Table 2. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Gender into English 035 



Tested 

Gender Population 



Female 


337 


54.6% 


Male 


280 


45,4% 


TOTAL 


617 


100.0% 



Recommended 


% Points 


Eligible/ 


to Enroll 


Difference 


Tested 


41 49.4% 


-5.2% 


.1217 


42 50.6% 


+5.2% 


.1500 


83 100.0% 







80% Rule Index: 123.3% 
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Table 3. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Gender into English 090 



Gender 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Female 


337 54.6% 


176 


55.0% 


+0.4% 


.5223 


Male 


280 45.4% 


144 


45,0% 


-0.4% 


.5143 


TOTAL 


617 100.0% 


320 


100.0% 







80% Rule Index: 98.5% 



Table 4. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Gender into English 101 



Gender 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Dilference 


Eligible/ 

Tested 


Female 


337 54.6% 


105 


56.8% 


+2.2% 


.3116 


Male 


280 45.4% 


80 


43.2% 


-2.2% 


.2857 


TOTAL 


617 100.0% 


185 


100.0% 







80% Rule Index; 91.7% 
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Table 5. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Learning Disability into English Oil 

80% Rule Index; Non-Disabled used as majority group 



Learninu Disability 


T ested 
Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Yes 


30 


4.9% 


3 


13.0% 


+8.1% 


.1000 


No 


577 


93.5% 


20 


87.0% 


-6.5% 


.0347 


No response 


10 


1.6% 


0 


0.0% 


-1.6% 




TOTAL 


617 


100.0% 


23 


100.0% 







80% Rule Index - 288.2% 



Table 6. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Learning Disability into English 035 



Learninu Disability 


Tested 

PoDulation 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Yes 


30 


4.9% 


10 


12.0% 


+7.1% 


.3333 


No 


577 


93.5% 


71 


85.5% 


-8.0% 


.1230 


No response 


10 


1.6% 


2 


2.4% 


+0.8% 




TOTAL 


617 


100.0% 


83 


100.0% 







80% Rule Index - 270.9% 
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Table 7. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Learning Disability' into English 090 



Learnina Disability 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Yes 


30 


4.9% 


14 


4.4% 


-0.5% 


.4667 


No 


577 


93.5% 


299 


93.4% 


-0.1% 


.5182 


No response 


10 


1.6% 


7 


2.2% 


+0.6% 




TOTAL 


617 


100.0% 


320 


100.0% 







80% Rule Index - 90. 1% 



Tables. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Learning Disability into English 101 



Learning Disability 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible 

Tested 


Yes 


30 


4.9% 


2 


1.1% 


-3.8% 


.0667 


No 


577 


93.5% 


182 


98.4% 


. +4.9% 


.3154 


No response 


10 


1 .6% 


i 


0.5% 


-1.1% 




TOTAL 


617 


100.0% 


185 


100.0% 







80% Rule Index -21.1% 
Cell size too small. 
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Table 9. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Age into English Oil 

80% Rule Index: “18-24 year-olds” used as majority group 



Age 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Under 1 8 


109 


17.7% 


1 


4.3% 


-13.4% 


.0092 


18-24 


334 


54.1% 


10 


43.5% 


-10.6% 


.0299 


25 and over 


174 


28.2% 


12 


52,2% 


+24.0% 


.0690 


TOTAL 


617 


100.0% 


23 


100.0% 







80% Rule for “Under 18" - 30.8% 
Cell size too small 

80% Rule for “25 and over” - 230.8% 



Table 10. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Age into English 035 



Age 


Tested 

PoDulation 


Recommended 
to Enroll 


% Points - 
Difference 


Eligible/ 

Tested 


Under 18 


109 


1 7.7% 


8 


9.6% 


-8.1% 


.0734 


18-24 


334 


54.1% 


44 


53.0% 


-1.1% 


.1317 


25 and over 


m 


28.2% 


2i 


37,3% 


+9.1% 


.1782 


TOTAL 


617 


100.0% 


83 


100.0% 







80% Rule for “Under 18" - 55.7% 
Cell size too small 

80% Rule for “25 and over” - 135.3% 




Pin i 
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Table 11. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Age into English 090 



Age 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Under 18 


109 


17.7% 


63 


19.7% 


+2.0% 


.5780 


18-24 


334 


54.1% 


184 


57.5% 


+3.4% 


.5509 


25 and over 


UA 


28.2% 


73 


22.8% 


-5.4% 


.4195 


TOTAL 


617 


100.0% 


320 


100.0% 







80% Rule for “Under 18" - 104.9% 
80% Rule for “25 and over” - 76.1% 



Table 12. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Age into English 101 



Age 


Tested 

PoDulation 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


Under 18 


109 


'17.7% 


37 


20.0% 


+2.3% 


.3394 


18-24 


334 


54.1% 


94 


50.8% 


-3.3% 


.2814 


25 and over 


174 


28,2% 


M 


29,2% 


+ 1.0% 


.3103 


TOTAL 


617 


100.0% 


185 


100.0% 







80% Rule for “Under 18" - 120.6% 
80% Rule for “25 and over” - 1 10.3% 
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Table 13. 

Wriling Sample Test 

Students Assessed and Recouimeiided to Enroll 
by Race/Etlinicit}' into English Oil 

80% Rule Index: Whites used as majority group 



Tested 

Population 

8 1.3% 



Recommended 
to Enroll 

0 0 . 0 % 



Race/Ethnicity 

American Indian/ 

Alaskan Native 

Asian/Pacific Islander/ 54 



Filipino 

Black/Non-Hispanic 24 

Hispanic 144 

White/Non-Hispanic 351 

Other 26 

No Response 10 

* 

TOTAL ' 617 



8.8% 


4 


\1A% 


3.9% 


1 


4.3% 


23.3% 


12 


52.2% 


56.9% 


5 


21.7% 


4.2% 


1 


4.3% 


1 ,6% 


0 


0,0% 


100.0% 


23 


100.0% 



% Points 
Difference 


Eligible/ 

Tested 


-1.3% 


.0000 


+8.6% 


.0741 


+0.4% 


.0417 


+28.9% 


.0833 


-35.2% 


.0142 


+0.1% 


.0385 


-1.6% 


_ ^ 



80% Rule for American Indian - no data 
80% Rule for Asian/Pacific Islander/Filipino - 521.8% 
80% Rule for Black - 293.7% 

80% Rule for Hispanic - 586.6% 

80% Rule for Other - 271.1% 
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Table 14. 

Writing Sample Test 

Students Assessed and Reconinieiided to Enroll 
by Race/Ethnicity into English 035 



Race/Ethnicitv 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


American Indian/ 


8 


1.3% 


2 


2.4% 


+1.1% 


.2500 


Alaskan Native 


Asian/Pacific Islander/ 


54 


8.8% 


10 


12.0% 


+3.2% 


.1852 


Filipino 


Black/Non-liispanic 


24 


3.9% 


5 


6.0% 


+2.1% 


.2083 


Hispanic 


144 


23.3% 


30 


36.1% 


+ 12.8% 


.2083 


White/Non-Hispanic 


351 


56.9% 


33 


39.8% 


-17.1% 


.0940 


Other 


26 


4.2% 


0 


0.0% 


-4.2% 


.0000 


No Response 


iO 


1 .6% 


3 


3.6% 


+2.0% 


— 


TOTAL 


617 


100.0% 


83 


100.0% 







80% Rule for American Indian/Alaskan Native - 266.0% 
80% Rule for Asian/Pacific Islander/Filipino - 197.0% 
80% Rule for Black - 221 .6% 

80% Rule for Hispanic - 22 1 .6% 

80% Rule for Other - no data 
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Table 15. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Race/Etlinicit)' into English 090 



Race/Ethnicitv 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


American Indian/ 


8 


1.3% 


4 


1.3% 


0.0% 


.5000 


Alaskan Native 


Asian/Pacific Islander/ 


54 


8.8% 


25 


7.8% 


-1.0% 


.4630 


Filipino 


Black/N on-Hispanic 


24 


3.9% 


13 


4.1% 


+0.2% 


.5417 


Hispanic 


144 


23.3% 


67 


20.9% 


-2.4% 


.4653 


WNte/Non-Hispanic 


351 


56.9% 


194 


60.6% 


+3.7% 


.5527 


Other 


26 


4.2% 


14 


4.4% 


+0.2% 


.5385 


No Response 


10 


1 ,6% 


3 


0.9% 


-0.7% 


— 


TOTAL 


617 


100.0% 


320 


100.0% 







80% Rule for American Indian/ Alaskan Native - 90.5% 
80% Rule for Asian/Pacific Islander/Filipino - 83.8% 
80% Rule for Black -98.0% 

80% Rule for Hispanic - 84.2% 

80% Rule for Other - 97.4% 
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Table 16. 

Writing Sample Test 

Students Assessed and Recommended to Enroll 
by Race/Etlinicity into English 101 



Race/Etlmicitv 


Tested 

Population 


Recommended 
to Enroll 


% Points 
Difference 


Eligible/ 

Tested 


American Indian/ 


8 


1.3% 


2 


1.1% 


-0.2% 


.2500 


/Maskan Native 


Asian/Pacific Islander/ 


54 


8.8% 


13 


7.0% 


-1.8% 


.2407 


Filipino 


Black/Non-Hispanic 


24 


3.9% 


5 


2.7% 


-1.2% 


.2083 


Hispanic 


144 


23.3% 


31 


16.8% 


-6.5% 


.2153 


White/Non-Hispanic 


351 


56.9% 


119 


64.3% 


+7.4% 


.3390 


Other 


26 


4.2% 


11 


5.9% 


+ 1.7% 


.4231 


No Response 


JO 


1.6% 


4 


2.1% 


+0.5% 


— 


TOTAL 


617 


100.0% 


185 


100.0% 







80% Rule for American Indian/Alaskan Native - 73.7% 
Cell size too small 

80% Rule for Asian/Pacific Islander/Filipino - 71.0% 
Cell size too small 

80% Rule for Black - 61.4% 

Cell size too small 

80% Rule for Hispanic - 63.5% 

80% Rule for Other - 124.8% 
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